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In many large computer systems with real-time use (such as the No. 1 
Electronic Switching Sy stein), the central processing unit handles much 
of its work through queues. It may spend much of its time cycling through 
the queues, performing the work requests it finds there. To accomodate 
varying degrees of urgency, the cycle may visit some hoppers more often 
than others. (No. 1 ESS strongly relies on this procedure.) This paper 
provides an approximate method for evaluating different cycles. 

Using the evaluation method and some approximations, we obtain a 
formula for the optimum relative frequency with which different queues 
should be visited. 

The model used is nonprobabilistic, and treats requests as continuous 
rather than discrete. The model also ignores certain interdependencies 
between queues. Despite these drastic simplifications, the results probably 
provide useful guidance, if interpreted cautiously. 

I. INTRODUCTION 

In many large computer systems, especially those with real-time 
use, the central processor handles much of its work through queues, 
which contain work requests. (The queues may also be called hoppers, 
buffers, waiting lines, files, and so forth. In this paper we call them 
hoppers.) The processor examines each hopper in turn, and performs 
some or all of the work requests if any, which it finds there. 

Some work requests require processing more urgently than others. 
One method of providing appropriate response times is to examine 
more frequently hoppers which contain urgent work, and other hoppers 
less frequently. For example, the No. 1 ESS (Electronic Switching 
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System) has many hoppers which it groups into five different urgency 
classes. 1,2 The five classes are examined (or "visited") in a fixed re- 
curring cycle, of length 30, during which the classes are visited 15, 
8, 4, 2, and 1 times, respectively. (During a single visit to a single class, 
the individual hoppers are visited once each, in a fixed sequence.) 

This paper contains a practical approximate model for evaluating 
various alternative cycles. The conceptual basis for the evaluation 
is the expected time each work request must wait in the hopper before 
being serviced by the central processor. (Such times depend not only 
on the cycle, but also on the times required to process requests, and 
on the rates at which new requests are initiated. These are all assumed 
given.) The expected waiting times for different hoppers are multi- 
plied by frequencies and also by weights w t , the "average penalty 
per second of delay," and added. The resulting sum is called P, the 
"expected total penalty per second." The weights u\- , which reflect 
the relative importance of delaying different work requests, are assumed 
given, and we seek to minimize P by choosing the cycle wisely. By 
way of illustration, the calculations required to evaluate any given 
cycle are given for two very simple cycles. 

When applied to general cycles, our model yields the plausible 
conclusion that visits to the same hopper should be spaced as evenly 
as possible around the cycle (in terms of elapsed time between visits). 
Furthermore, the model permits us to estimate how sensitive P is to 
deviations from this ideal. 

Our most important conclusion is an explicit formula for how fre- 
quently each hopper should be visited. To obtain this formula, we 
assume that visits to each hopper are evenly spaced around the cycle. 
Then P becomes a function of the visit frequency (and not of detailed 
visit pattern). We explicitly optimize this function, to obtain a formula 
for visit frequencies. 

The time required to examine a hopper, whether or not it contains 
any work requests, is small but highly significant, and is an important 
consideration in the problem. Our model explicitly reflects this fact. 
(Indeed, it is known though sometimes overlooked that the No. 1 ESS 
central processor finds most hoppers empty on a majority of its visits, 
even when it is heavily loaded with work and operating near its ca- 
pacity limits. This can occur because the number of hoppers is so large, 
and because each work request requires a relatively long time to service 
compared with the time to visit a single hopper.) 

In this study, we assume that work enters the hoppers as a result 
of some outside process, which is independent of how the hoppers 
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are being served. In No. 1 ESS, as in many other situations, much 
work does enter hoppers in this manner. However, it is also true that 
servicing a request from one hopper may place work, directly or in- 
directly, in another hopper. This interdependence may well be important 
in choice of a cycle. Nevertheless, the present model, which ignores 
such interdependence, is probably usable if we are suitably cautious 
about interpreting our results. 

Service requests are discrete items and enter the hoppers according 
to an exceedingly complicated random process. Our model, however, 
assumes that each kind of request comes in at a constant rate, with 
no statistical fluctuation whatsoever. Furthermore, we treat the number 
of requests as a continuous quantity (so that requests keep trickling 
in like water) rather than a discrete quantity. 

Despite the drastic nature of all these simplifications, we believe 
that this analysis is better than no analysis at all. Furthermore, we 
feel that our conclusions are probably valid approximations. It also 
seems plausible that our model could provide the jumping-off place 
for a more realistic study. Both interdependence and statistical fluc- 
tuation could be introduced in a limited way. (Since this was first 
written, R. W. Landgraff has done a study which extends this model 
to include interdependence. 3 ) This might well permit their main effects 
to enter the model, without opening the Pandora's box of an ex- 
tremely general stochastic process with one server and many inter- 
dependent queues. 

II. SOME ASSUMPTIONS AND NOTATION 

We suppose that there are i" hoppers. For each hopper i we assume 
that we have three parameters: 

St = service time = average time to service one request in this 

hopper, 
r, = request time = average time between occurrence of requests 

» s if and 
10 i = weight = average penalty per second of delay for a single 

request of type i. 



We also use 



A,. = ^« 1, A = £\, . 
r, T^i 



(To permit a steady-state solution, we assume A < 1.) Note that the 
definition of w t implies that on the average the penalty for delaying 



2966 THE BELL SYSTEM TECHNICAL JOURNAL, NOVEMBER 1969 

one kind of task is proportional to the delay time. The w, are the pro- 
portionality constants. This simple assumption could be refined some- 
what without too much trouble if desired. 

In No. 1 ESS, one major penalty caused by hopper delays is the 
extra waiting time they cause to the telephone user at various stages 
of his call. For some hoppers, such as those involved during the process 
of dialing, undue delays can cause mishandling of the call. (Also, the 
delays tie up memory capacity and indirectly cause a need for extra 
memory equipment. However, this effect is probably minor.) By 
considering the loss incurred by the user due to various waiting periods, 
and the loss due to the probability of mishandled call, it would be 
possible to assign sensible values to the w { . Although a truly realistic 
appraisal of the losses would require a quite elaborate study, some 
fairly reasonable simplifying assumptions which would make this 
study much simpler are available. Furthermore, assignment of the 
Wi on a direct intuitive basis would probably be adequate for many 
purposes. 

To measure the total delay penalty paid by any work-scheduling 
algorithm, we combine the various penalties into a single number P: 

d { = expected delay for a request of type i, 

Pi = expected penalty per request of type i = to, d { , and 

P = expected total penalty per second 






(Of course, 1/r, is the expected number of requests of type i in one 
second.) We seek to minimize P by proper choice of a work-scheduling 
algorithm. Only the delays d, may be influenced in this way, so we 
concentrate on evaluating the d { . 

A model which, like ours, treats requests as continuous has the 
danger of "discovering" that the hoppers are serviced infinitely fast, 
accumulating only an infinitesimal amount of work between visits. 
The following assumption, which in any case reflects an important 
reality, avoids this collapse. 

To examine the ith hopper, whether or not it contains any work, 
requires a certain amount of time. We assume this amount of time 
is Hi . For simplicity we shall assume all the H , are equal, and shall call 
their common value H, although it would be easy to work with unequal 
values if desired. Thus if x requests are serviced during one visit to 
hopper i f this visit requires H t -j- xs, seconds. 
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It will turn out later that the value chosen for H is not very important 
in the context of this model. The comparison between different work- 
scheduling algorithms is unaffected by the (nonzero) value used. 

III. WORK-SCHEDULING AND SERVICE POLICY 

We suppose that the hoppers are visited in a fixed cycle of length 
N, namely, 

(*I i H i • ■ ■ ) w- 

This means that hopper i x is visited first, then hopper i 2 , and so on. 
After i N is visited, the cycle starts over again with hopper i s . One simple 
cycle with / = 4 and N = 6 is (1, 4, 2, 4, 3, 4). No. 1 ESS uses 1 = 5 
hoppers (classes of hoppers, actually), and a cycle of length N = 30: 

121312141213121512131214121312 

If i is any given hopper, we shall let V(i) indicate the set of all visits 
to hopper i. Thus for the cycle (1, 4, 2, 4, 3, 4), we have 

7(1) = [1], 7(2) = [3], 7(3) = [5], and 7(4) = [2, 4, 6]. 

In the No. 1 ESS cycle, 

7(1) = [1, 3, 5, • • • , 29], 7(2) = [2, 6, 10, 14, 18, 22, 26, 30], 

7(3) - [4, 12, 20, 28], 7(4) = [8, 24], 7(5) = [16]. 

For any visit n, the last previous visit to the same hopper is called 
b(n) ("b" for before). Thus in the cycle (1, 4, 2, 4, 3, 4), visit 6 is to 
hopper 4, and the last previous visit to the same hopper is on visit 4. 
Thus 6(6) = 4. Because "last previous" is understood in a cyclic sense, 
6(2) = 6. We have 

6(1) = 1, 6(2) = 6, 6(3) = 3, 

6(4) = 2, 6(5) = 5, 6(6) = 4. 

Whenever a hopper is visited, we suppose that all work requests 
there are serviced. However, during the period when the hopper is 
being serviced new requests can enter it. What about these requests 
which enter the hopper while it is being serviced? These can either be 
handled when they are reached during the same visit, which we call 
the "come-right-in" policy, or they can be left for the next visit to the 
hopper, which we call the "please-wait" policy. We shall treat both of 
these hopper service policies, because their solutions are very similar. 
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IV. HOW TO EVALUATE P 

As there is no statistical variation left in our model, it is easy to 
analyze. Let 

t n = time spent emptying the hopper i n during visit n. 

Let C be the time spent during an entire cycle, so that C consists of N 
hopper visits. Hopper visit n consists of time H to examine the hopper, 
and time t n to service it. Thus 

C - £ (H + Q = NH + £ «, . 

n-l n-1 

Now consider the requests which are serviced during t„ . Let 

T n = the interval during which they enter hopper i n . 

Recalling that b(ri) is the last prior visit to hopper i n , we see from 
Fig. 1 that 

E (H + U = [n - 6(n)]ff + E «, , 

6(n) + l ft(n)+l 

"come-right-in;" 
'/'„ - ( (1) 

E «, + B) = [n - Hn))H + £ «, , 

lp-i(n) K») 

"please-wait." 

Note that b(n) and the summation indices must be understood in a 
suitable "cyclic" sense, so that (for example) if b(ri) = n, then n — b(n) 
means once around the cycle and hence equals N, not 0. Now it is easy 
to see that 

(the number of requests served during t n ) = t„/s in 

= (the number of requests initiated during T„) = TJr in , 

T n FOR COME-RIGHT-IN 
J . 

i H , t m ! H , t m + , , | H , tn I 



t , 1 

m = b (n) T n p OR PLEASE-WAIT 

Fig. 1 —Time flow diagram illustrating "please-wait" and "come-right-in" policies. 
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SO 

(n = \ im T„ . (2) 

By using equation (2), we can eliminate either all T„ or all t n from equa- 
tions (1). This will leave us with N linear equations in N unknowns, 
which in fact turn out to be linearly independent. By solving these 
equations and using equation (2), we can find the T n and the t n , and 
from them all else will follow, as we show below. For convenient ref- 
erence, we state the equations after eliminating the t n : 



T n = < 



[n - b(n)]H + £ KT P , "come-right-in;" 

n-1 

[n - b(n)]H + £ \ ip T v , "please-wait." 



Recall the special cychc interpretation of n — b(n) and the summations. 
It is worth digressing briefly to derive an explicit formula for C, 
and to show how the N equations (3) can be reduced toN — I equations 
in N — I unknowns by using it. It is easy to see that if we sum T„ over 
all visits to some particular hopper j, the result must equal C: 

Y, T„ = C for every j. (4) 

n in K(») 

Now sum equation (3) over all n in V(j), and use equation (4) several 
times: 

£ T n = £ {[n - b(n)]H + £ Kt} , 

ninV(i) ninV(j) ^ p-6(n)+l ) 

C = NH + £ X,^ 

p-i 

= nh+J: x,[ S rj 



i-i 



= NH + £x,C 



!=1 



= NH + AC. 



This yields 



- - ft 
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Since C is now given directly in terms of known quantities, we can use 
equation (4) to solve for one T n in terms of others. We can do this 
separately for each j = 1 to I, and thereby reduce the number of un- 
knowns and equations to N-I. 

Once we have the values of T n (and hence of t n ), we may easily 
evaluate e n , the average delay for requests serviced during visit n. 
(Each delay is reckoned from occurrence of request to when its proc- 
essing starts.) By elementary reasoning, we see that 

e n = ICT, - U, "come-right-in," I 



e„ = h(T n + Q, "please-wait." 

Of course T n /r in requests are serviced in visit n. Thus the average 
delay per request of type j is 

E T f, 

d, = ^-^~ (7) 

£ f 

n in VU) I i 

Using equations (6), (2), and (4), we get 

d t = ^=7^ E Tl , "come-right-in," 

^k n in VU) 

d< = ^-^ E Tl , "please-wait." 



in f(f) 



(8) 



Now let 



so that 



Then 



F n = TJC = fraction of a cycle used by T n , 

2 F. -1, alii. (9) 



v<o 



fi(l - X,)C X) ft , "come-right-in," 

d,. = ^ niDTr( ° (10) 

[same, but with 1 + X< for 1 — X, , "please-wait." 

Using equation (5) and the definition of P, we now easily find a 
formula for the penalty P, which is the key quantity we use to evaluate 
work-scheduling algorithms: 
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P = . 



NH ^VHin_x) Y F 2 \ " come - ri g ht - in >" 



2(1 - A) IV r, nlnVtn j 

same, but with 1 + X, for 1 — X, , 

1 — A is unaffected "please- wait." 



(H) 



(However, note that the values of the F n may differ for the two policies.) 
We note that the work-scheduling algorithm influences equation (11) 
in only two ways: through N, and through the fractions F„ . From 
this formula we can evaluate and compare different work-scheduling 
algorithms. Also we can compare "come-right-in" with "please-wait." 

V. SOME EXAMPLES 

If there are 7 = 3 different hoppers, the simplest possible cycle is 
(1, 2, 3), for which N = 3. In this case we see trivially that F, = F 2 = 
F 3 = 1, for either "come-right-in "or "please-wait." Thus equation 
(10) for cycle (1, 2, 3) is: 



f 3// 



P = 



52 — (1 - X,), "come-right-in," 



2(1 - A) V i\ 

same, but with 1 + X, for 1 — X, , 

1 — A is unaffected "please-wait." 

Given the three input parameters s, , r, , and w;, for each hopper, 
this can be evaluated numerically. 

Now suppose we use the cycle (1, 2, 1, 3), for which N = 4, with 
the "come-right-in" policy. Then equations (3) for cycle (1, 2, 1, 3) 
become the following four equations: 

7\ = X^, + x 3 r 4 + 2H, 

T 2 - K(T t + T 3 ) + X 2 T 2 + X 3 T 4 + 4//, 

r, - x 2 r 2 + kt 3 + 2H, 

T, = X,(7\ + T 3 ) + \ 2 T 2 + \ 3 Ti + 4H. 

However, taking C as known, and using equation (4) for cycle (1,2, 1,3) 
namely, 

7\ + T 3 = C, T 2 = C, T t = C, 
we eliminate the unknowns T 2 , T 3 , and T 4 , leaving one equation in 
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one unknown, 7\ : 

T x = X,T, + X 3 C + 2H. 
We find 

T x = I -^-[2H + X 3 C]. 

Dividing by C, and using C = 4#/(l — A) from equation (5), we see 

_ j(l - A) + X 3 _ 1 1 - X, + X 3 - X 2 
tl 1 - X, 2 1 - X, 



As F :i m 1 — Fi , we find 



it+& 



FI + fl - - ' - % ' V 



2 L \i - x, / _ ' 
and also 

n = i, *? = i. 

Thus equation (11) for cycle (1, 2, 1, 3) with the "come-right-in" 
policy is 

+ ^ (1 _ X2) + S (1 _x 3) 

Through special circumstances which would not hold in general, the 
values for F„ using this cycle are all the same for "please-wait" as 
for "come-right-in," so P for "please-wait" is the same as the above 
but with 1 + X, substituted for 1 -f- X,- in three places. Given the 
parameters s, , r, , and w t for each hopper, this can be evaluated 
numerically. 

VI. CONCLUSIONS 

If we compare cycles of the same length and with the same number 
of visits to each hopper, then equation (11) yields the following con- 
clusion: The visits to a given hopper should be spaced as evenly around 
the cycle as possible. 
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By this we mean that the values of T n (and hence of F n ) pertaining 
to this hopper should be as equal as possible. This follows because 
the minimum of 

£ Fl subject to £ F « = 1 

■ il71(] ninV(.) 

occurs when the F n with n in V(i) are all equal. Furthermore, equation 
(11) can be used to estimate how serious any given deviation from 
equality is. 

Suppose a cycle has N t visits to hopper i, so that N = SiV,- , and 
suppose that the N { visits are spaced approximately evenly around 
the cycle for every i. Then for each visit n to hopper *, 



K ™w; 



Thus 



Either using a Lagrange multiplier to handle the constraint that 
SiV,- = N, or by direct argument (see the appendix), it is easy to deduce 
that the values of iV, which minimize this satisfy 



N,- proportional to — '- (1 — 



fe*-»r 



so 



N 



^(i-x,)] 1 ' 



-Ti 



-i-x,)] 



r,- 



This yields our most important conclusion: The above approximate 
formula gives the optimum relative frequency of visits to each hopper in 
the cycle. 

By obtaining values for r,- , s { , and less easily for w< , it is possible 
to compare different work-scheduling algorithms with each other and 
with the "ideal" schedule with perfect spacing implied above. Notice 
that the actual value of H does not enter into this comparison. (If we 
had used unequal values for the Hi , only the ratios Hi/Hj would 
enter into the comparison, not the actual values of the Hi themselves.) 
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It would probably be worthwhile to analyze the actual work-sched- 
uling algorithm used for ESS No. 1 in these terms. It would be in- 
teresting to compare this actual algorithm with the "ideal" algorithm. 

Our model, with its highly simplified assumptions, cannot possibly 
provide the last word on work-scheduling evaluations, even with regard 
to delay times. However, this kind of approach is probably desirable. 
If greater realism is desired, the most important aspects are statistical 
variability and interdependence of hoppers. 

APPENDIX 

Direct Argument to Replace the LaGrange Multiplier Argument 

Henry Pollak has pointed out a simple direct argument which shows 
that 2(a,-/iV,-) is minimized, subject to the constraint ZiV< = N, if 
Ni is proportional to (a,-)*. Using a, = «\(1 — A,)/?-,- , this yields the 
formula given above for N { . 

First, let q = 2V/[2(a,-)*]. Now, we multiply the quantity to be min- 
imized by q\ and express it: 

The middle term is constant by definition, and the last term is con- 
stant by constraint. The first term cannot be less than 0. The first 
term is if 

&■ = N< or N t = g(a,) 1/a . 

Since these values satisfy the constraint, we obtain the desired result. 
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